Add .NET refactoring skill by AbhitejJohn · Pull Request #873 · dotnet/skills

AbhitejJohn · 2026-07-08T17:09:16Z

Summary

Refactoring is a common operation for .NET developers, and a dedicated skill can help agents make these changes in the way .NET teams expect: behavior-preserving, incremental, and validated with build/test gates.

This PR adds two related skills:

csharp-refactoring: guides safe C#/.NET refactoring work such as rename, move, extract, inline, split, consolidate/de-duplicate, and modernization while preserving behavior.
dotnet-breaking-changes: provides the compatibility guardrails needed when a refactor touches public API, multi-targeting, source-generated/partial code, or InternalsVisibleTo surfaces.

Why these skills

csharp-refactoring focuses on the refactoring workflow itself: establish a green baseline, choose one named operation, find true references, prefer semantics-aware edits, and re-gate after each step. Its reference file expands the operation catalog and maps common refactoring operations to the kinds of changes .NET teams regularly make.

dotnet-breaking-changes covers the .NET-specific surfaces where a change can compile and pass tests while still breaking downstream consumers. Its reference files explain how to inspect and handle:

public API gates (PublicAPI.*, ApiCompat, package validation)
multi-targeting and #if branches
source-generated and partial code
friend assemblies via InternalsVisibleTo

Together, the skills let an agent keep refactoring focused on behavior preservation while still checking the .NET compatibility surfaces that matter in real repos.

Validation

Added eval coverage for both skills.
Included fixture scenarios that exercise rename, extraction, consolidation, public API checks, multi-targeting, generated/partial code, and friend assembly hazards.
Ran the skill validator successfully.
Verified the eval fixtures build and test successfully in isolated copies.

github-actions · 2026-07-08T17:09:50Z

Skill Coverage Report

	Plugin	Skill	Covered	Coverage
⚠️	`dotnet`	`setup-local-sdk`	4/8	50%

Uncovered: dotnet/setup-local-sdk

[Pitfall] dotnet app.dll wrong runtime (line 386)
[CodePattern] [guid] (line 104)
[CodePattern] [ordered] (line 301)
[CodePattern] [pscustomobject] (line 301)

AbhitejJohn · 2026-07-08T17:27:07Z

/evaluate

github-actions · 2026-07-08T17:53:13Z

Skill Validation Results

Skill	Scenario	Quality	Skills Loaded	Overfit	Verdict
csharp-refactoring	Rename a method across its declaration and every caller	3.0/5 → 1.7/5 🔴	✅ csharp-refactoring; tools: skill, glob / ✅ csharp-refactoring; tools: skill, read_bash, glob	✅ 0.16	❌ [1]
csharp-refactoring	Extract a repeated calculation into a private helper	2.0/5 → 1.7/5 🔴	✅ csharp-refactoring; tools: grep, skill / ⚠️ NOT ACTIVATED	✅ 0.16	❌ [2]
csharp-refactoring	Consolidate a duplicated block into a single shared helper	1.3/5 → 1.0/5 🔴	✅ csharp-refactoring; tools: skill, glob / ⚠️ NOT ACTIVATED	✅ 0.16	❌ [3]
csharp-refactoring	Inline a pass-through wrapper and update callers	2.7/5 → 1.0/5 🔴	✅ csharp-refactoring; tools: skill, bash / ⚠️ NOT ACTIVATED	✅ 0.16	❌ [4]
csharp-refactoring	Merge two near-identical types into one parameterized type	1.0/5 → 1.0/5	⚠️ NOT ACTIVATED / ✅ csharp-refactoring; tools: bash, read_bash, glob, edit, skill	✅ 0.16	❌ [5]
csharp-refactoring	Decline a framework and package upgrade dressed up as a refactor	1.3/5 → 1.0/5 ⏰ 🔴	ℹ️ not activated (expected)	✅ 0.16	❌ [6]
csharp-refactoring	Decline a new-feature request dressed up as a refactor	4.7/5 → 3.0/5 ⏰ 🔴	ℹ️ not activated (expected)	✅ 0.16	✅ [7]
csharp-refactoring	Keep behavior-changing bug fixes out of a behavior-preserving refactor	3.7/5 ⏰ → 2.3/5 ⏰ 🔴	ℹ️ not activated (expected)	✅ 0.16	❌ [8]
dotnet-breaking-changes	Answer a public-API question before consolidating duplicated parsing	2.0/5 → 1.3/5 🔴	✅ dotnet-breaking-changes; tools: skill, glob / ⚠️ NOT ACTIVATED	✅ 0.10	✅ [9]
dotnet-breaking-changes	Add behavior to a shipped public member without breaking the contract	3.3/5 → 2.3/5 🔴	✅ dotnet-breaking-changes; tools: skill / ⚠️ NOT ACTIVATED	✅ 0.10	❌ [10]
dotnet-breaking-changes	Rename a helper referenced from every #if branch of a multi-targeted type	3.0/5 → 3.0/5	✅ dotnet-breaking-changes; tools: skill / ⚠️ NOT ACTIVATED	✅ 0.10	✅ [11]
dotnet-breaking-changes	Rename a member of a partial type that a generated part references	3.0/5 → 2.0/5 🔴	✅ dotnet-breaking-changes; tools: skill, glob / ⚠️ NOT ACTIVATED	✅ 0.10	❌ [12]
dotnet-breaking-changes	Add a new public method that must be correct on every target framework	2.0/5 → 1.0/5 🔴	✅ dotnet-breaking-changes; tools: bash, glob, skill / ⚠️ NOT ACTIVATED	✅ 0.10	❌ [13]
dotnet-breaking-changes	Do not raise breaking-change concerns for a purely internal local rename	5.0/5 → 5.0/5	ℹ️ not activated (expected)	✅ 0.10	❌ [14]

[1] ⚠️ High run-to-run variance (CV=82%) — consider re-running with --runs 5
[2] ⚠️ High run-to-run variance (CV=87%) — consider re-running with --runs 5
[3] ⚠️ High run-to-run variance (CV=109%) — consider re-running with --runs 5
[4] ⚠️ High run-to-run variance (CV=58%) — consider re-running with --runs 5
[5] ⚠️ High run-to-run variance (CV=265%) — consider re-running with --runs 5. (Isolated) Quality unchanged but weighted score is -15.5% due to: judgment, quality
[6] ⚠️ High run-to-run variance (CV=71%) — consider re-running with --runs 5
[7] ⚠️ High run-to-run variance (CV=122%) — consider re-running with --runs 5. (Isolated) Quality dropped but weighted score is +4.0% due to: tokens (180608 → 105541), tool calls (19 → 11), time (211.0s → 136.3s)
[8] ⚠️ High run-to-run variance (CV=114%) — consider re-running with --runs 5
[9] ⚠️ High run-to-run variance (CV=159%) — consider re-running with --runs 5. (Isolated) Quality dropped but weighted score is +15.2% due to: completion (✗ → ✓), tokens (90688 → 78437)
[10] ⚠️ High run-to-run variance (CV=171%) — consider re-running with --runs 5
[11] ⚠️ High run-to-run variance (CV=105%) — consider re-running with --runs 5
[12] ⚠️ High run-to-run variance (CV=103%) — consider re-running with --runs 5
[13] ⚠️ High run-to-run variance (CV=56%) — consider re-running with --runs 5
[14] ⚠️ High run-to-run variance (CV=106%) — consider re-running with --runs 5

⏰ timeout — run(s) hit the (300s) scenario timeout limit; scoring may be impacted by aborting model execution before it could produce its full output (increase via timeout in eval.yaml)

Model: claude-opus-4.6 | Judge: claude-opus-4.6

🔍 Full Results - additional metrics and failure investigation steps

To investigate failures, paste this to your AI coding agent:

For PR 873 in dotnet/skills, download eval artifacts with gh run download 28962446218 --repo dotnet/skills --pattern "skill-validator-results-*" --dir ./eval-results, then fetch https://raw.githubusercontent.com/dotnet/skills/1b3a5bd13db825237d579a4dd9d2c1dfc87d9953/eng/skill-validator/src/docs/InvestigatingResults.md and follow it to analyze the results.json files. Diagnose each failure, suggest fixes to the eval.yaml and skill content, and tell me what to fix first.

▶ Sessions Visualisation -- interactive replay of all evaluation sessions
📊 Session Analytics (preview) -- aggregated metrics across evaluation sessions

Copilot

Pull request overview

This PR adds two new .NET-focused skills to the plugins/dotnet plugin—one for behavior-preserving C# refactoring workflows and one for .NET-specific breaking-change/compatibility guardrails—along with eval coverage and fixtures that exercise the key hazards (public API baselines, multi-targeting #if, generated/partial code, and friend assemblies).

Changes:

Added csharp-refactoring skill guidance plus an operation catalog reference.
Added dotnet-breaking-changes skill guidance plus focused reference docs for the four “hidden surfaces”.
Added eval suites + identical lightweight Billing fixture solutions under tests/dotnet/ for both skills; updated CODEOWNERS and dotnet plugin README.

Show a summary per file

File	Description
tests/dotnet/dotnet-breaking-changes/tests/Billing.Tests/BillingTests.cs	Adds fixture tests used by the breaking-change eval scenarios.
tests/dotnet/dotnet-breaking-changes/tests/Billing.Tests/Billing.Tests.csproj	Adds an xUnit test project targeting net10.0 for the breaking-change fixture solution.
tests/dotnet/dotnet-breaking-changes/src/Billing/PublicAPI.Shipped.txt	Adds a public API baseline file used by eval prompts/assertions.
tests/dotnet/dotnet-breaking-changes/src/Billing/Pricing.cs	Adds pricing/tax types used for refactor/breaking-change exercises.
tests/dotnet/dotnet-breaking-changes/src/Billing/PlatformInfo.cs	Adds multi-targeting `#if` example surface for breaking-change checks.
tests/dotnet/dotnet-breaking-changes/src/Billing/OrderProcessor.cs	Adds intentionally-refactorable implementation used by scenarios.
tests/dotnet/dotnet-breaking-changes/src/Billing/Coupons.g.cs.template	Adds generator-input template to simulate generated/partial hazards.
tests/dotnet/dotnet-breaking-changes/src/Billing/Coupons.cs	Adds hand-authored partial type paired with the generated template.
tests/dotnet/dotnet-breaking-changes/src/Billing/Billing.csproj	Adds multi-targeted library project and build-time generation hook.
tests/dotnet/dotnet-breaking-changes/src/Billing/AppSettingsHelper.cs	Adds duplicated parsing helpers for public-API consolidation scenarios.
tests/dotnet/dotnet-breaking-changes/Fixture.sln	Adds the breaking-change fixture solution container.
tests/dotnet/dotnet-breaking-changes/eval.yaml	Adds breaking-change eval scenarios and build/test gates.
tests/dotnet/csharp-refactoring/tests/Billing.Tests/BillingTests.cs	Adds fixture tests used by the refactoring eval scenarios.
tests/dotnet/csharp-refactoring/tests/Billing.Tests/Billing.Tests.csproj	Adds an xUnit test project targeting net10.0 for the refactoring fixture solution.
tests/dotnet/csharp-refactoring/src/Billing/PublicAPI.Shipped.txt	Adds a public API baseline file used by refactoring prompts/assertions.
tests/dotnet/csharp-refactoring/src/Billing/Pricing.cs	Adds pricing/tax types used for refactoring exercises.
tests/dotnet/csharp-refactoring/src/Billing/PlatformInfo.cs	Adds multi-targeting `#if` example surface for refactoring checks.
tests/dotnet/csharp-refactoring/src/Billing/OrderProcessor.cs	Adds intentionally-refactorable implementation used by scenarios.
tests/dotnet/csharp-refactoring/src/Billing/Coupons.g.cs.template	Adds generator-input template to simulate generated/partial hazards.
tests/dotnet/csharp-refactoring/src/Billing/Coupons.cs	Adds hand-authored partial type paired with the generated template.
tests/dotnet/csharp-refactoring/src/Billing/Billing.csproj	Adds multi-targeted library project and build-time generation hook.
tests/dotnet/csharp-refactoring/src/Billing/AppSettingsHelper.cs	Adds duplicated parsing helpers used in refactoring scenarios.
tests/dotnet/csharp-refactoring/Fixture.sln	Adds the refactoring fixture solution container.
tests/dotnet/csharp-refactoring/eval.yaml	Adds refactoring eval scenarios and build/test gates.
plugins/dotnet/skills/dotnet-breaking-changes/SKILL.md	Introduces the breaking-change skill and when/when-not-to-use guidance.
plugins/dotnet/skills/dotnet-breaking-changes/references/source-generation.md	Adds detailed guidance for generated/partial code hazards.
plugins/dotnet/skills/dotnet-breaking-changes/references/public-api.md	Adds detailed guidance for public API gating and compatibility decisions.
plugins/dotnet/skills/dotnet-breaking-changes/references/multi-targeting.md	Adds detailed guidance for multi-targeting and `#if` branch correctness.
plugins/dotnet/skills/dotnet-breaking-changes/references/internals-visible-to.md	Adds detailed guidance for friend-assembly/internal-surface hazards.
plugins/dotnet/skills/csharp-refactoring/SKILL.md	Introduces the refactoring skill and a stepwise safety contract.
plugins/dotnet/skills/csharp-refactoring/references/operation-catalog.md	Adds a more detailed refactoring operation taxonomy reference.
plugins/dotnet/README.md	Updates the dotnet plugin skill list to include the new skills.
eng/known-domains.txt	Adds `github.com/dotnet/skills` to known domains.
.github/CODEOWNERS	Adds ownership entries for the new skills and their tests.

Copilot's findings

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Files reviewed: 34/34 changed files
Comments generated: 0

AbhitejJohn · 2026-07-13T17:23:19Z

/evaluate

github-actions · 2026-07-13T17:58:33Z

👋 @AbhitejJohn — this PR has 1 unresolved review thread(s). When you're ready, please address the feedback and push an update; the triage bot will pick up the next state automatically. (Add the no-stale label to silence further pings.)

github-actions · 2026-07-13T18:00:44Z

Skill Validation Results

Skill	Scenario	Quality	Skills Loaded	Overfit	Verdict
csharp-refactoring	Rename a method across its declaration and every caller	3.0/5 → 1.0/5 🔴	✅ csharp-refactoring; tools: skill, glob / ✅ csharp-refactoring; tools: skill, read_bash, glob, stop_bash	✅ 0.17	❌
csharp-refactoring	Extract a repeated calculation into a private helper	2.0/5 → 1.7/5 🔴	✅ csharp-refactoring; tools: grep, skill / ✅ csharp-refactoring; tools: grep, read_bash, skill	✅ 0.17	✅ [1]
csharp-refactoring	Consolidate a duplicated block into a single shared helper	2.0/5 → 1.7/5 🔴	✅ csharp-refactoring; tools: glob, skill, bash / ⚠️ NOT ACTIVATED	✅ 0.17	❌ [2]
csharp-refactoring	Inline a pass-through wrapper and update callers	2.3/5 → 1.0/5 🔴	✅ csharp-refactoring; tools: skill, bash / ⚠️ NOT ACTIVATED	✅ 0.17	❌
csharp-refactoring	Merge two near-identical types into one parameterized type	1.3/5 → 1.0/5 🔴	✅ csharp-refactoring; tools: skill, edit / ⚠️ NOT ACTIVATED	✅ 0.17	❌ [3]
csharp-refactoring	Decline a framework and package upgrade dressed up as a refactor	1.0/5 → 1.0/5	ℹ️ not activated (expected)	✅ 0.17	❌ [4]
csharp-refactoring	Decline a new-feature request dressed up as a refactor	4.0/5 → 5.0/5 🟢	ℹ️ not activated (expected)	✅ 0.17	❌ [5]
csharp-refactoring	Keep behavior-changing bug fixes out of a behavior-preserving refactor	2.3/5 ⏰ → 3.3/5 ⏰ 🟢	ℹ️ not activated (expected)	✅ 0.17	❌ [6]
dotnet-breaking-changes	Answer a public-API question before consolidating duplicated parsing	2.3/5 → 2.3/5	✅ dotnet-breaking-changes; tools: glob, skill / ⚠️ NOT ACTIVATED	✅ 0.12	✅ [7]
dotnet-breaking-changes	Add behavior to a shipped public member without breaking the contract	2.7/5 → 2.3/5 🔴	⚠️ NOT ACTIVATED	✅ 0.12	❌ [8]
dotnet-breaking-changes	Rename a helper referenced from every #if branch of a multi-targeted type	3.3/5 → 2.7/5 🔴	✅ dotnet-breaking-changes; tools: skill / ⚠️ NOT ACTIVATED	✅ 0.12	❌ [9]
dotnet-breaking-changes	Rename a member of a partial type that a generated part references	1.7/5 → 1.7/5	⚠️ NOT ACTIVATED	✅ 0.12	❌ [10]
dotnet-breaking-changes	Add a new public method that must be correct on every target framework	2.0/5 → 2.0/5	⚠️ NOT ACTIVATED	✅ 0.12	❌ [11]
dotnet-breaking-changes	Do not raise breaking-change concerns for a purely internal local rename	4.7/5 → 5.0/5 🟢	ℹ️ not activated (expected)	✅ 0.12	✅ [12]

[1] ⚠️ High run-to-run variance (CV=214%) — consider re-running with --runs 5. (Isolated) Quality dropped but weighted score is +1.7% due to: tokens (79654 → 66925), time (82.4s → 64.0s)
[2] ⚠️ High run-to-run variance (CV=177%) — consider re-running with --runs 5
[3] ⚠️ High run-to-run variance (CV=895%) — consider re-running with --runs 5
[4] ⚠️ High run-to-run variance (CV=98%) — consider re-running with --runs 5. (Isolated) Quality unchanged but weighted score is -17.3% due to: judgment, quality
[5] ⚠️ High run-to-run variance (CV=379%) — consider re-running with --runs 5. (Isolated) Quality improved but weighted score is -0.9% due to: tokens (151034 → 171934)
[6] ⚠️ High run-to-run variance (CV=404%) — consider re-running with --runs 5. (Plugin) Quality improved but weighted score is -71.1% due to: quality, judgment, tokens (212959 → 255721)
[7] ⚠️ High run-to-run variance (CV=337%) — consider re-running with --runs 5
[8] ⚠️ High run-to-run variance (CV=1202%) — consider re-running with --runs 5
[9] ⚠️ High run-to-run variance (CV=97%) — consider re-running with --runs 5
[10] ⚠️ High run-to-run variance (CV=339%) — consider re-running with --runs 5. (Isolated) Quality unchanged but weighted score is -48.4% due to: judgment, quality
[11] ⚠️ High run-to-run variance (CV=213%) — consider re-running with --runs 5. (Isolated) Quality unchanged but weighted score is -15.4% due to: judgment, quality
[12] ⚠️ High run-to-run variance (CV=215%) — consider re-running with --runs 5

⏰ timeout — run(s) hit the (300s, 600s) scenario timeout limit; scoring may be impacted by aborting model execution before it could produce its full output (increase via timeout in eval.yaml)

Model: claude-opus-4.6 | Judge: claude-opus-4.6

🔍 Full Results - additional metrics and failure investigation steps

To investigate failures, paste this to your AI coding agent:

For PR 873 in dotnet/skills, download eval artifacts with gh run download 29270232281 --repo dotnet/skills --pattern "skill-validator-results-*" --dir ./eval-results, then fetch https://raw.githubusercontent.com/dotnet/skills/a5265d860095cbd1b989dcaa451c1716de027d77/eng/skill-validator/src/docs/InvestigatingResults.md and follow it to analyze the results.json files. Diagnose each failure, suggest fixes to the eval.yaml and skill content, and tell me what to fix first.

▶ Sessions Visualisation -- interactive replay of all evaluation sessions
📊 Session Analytics (preview) -- aggregated metrics across evaluation sessions

JanKrivanek

Since the skills have nontrivial size - the main concern now is overlap between the two skills + cross-loading cost. csharp-refactoring instructs the agent to also load dotnet-breaking-changes, and the two share material ("stop and escalate," public API, forwarders, partial/generated). The split is defensible (breaking-changes is meant to apply to features/fixes too, not just refactors), but loading two long skills for one refactor is a real context cost that the evals don't yet justify.

Trim study for PR #873: csharp-refactoring-trim is a ~30% shorter rewrite of csharp-refactoring (drops When-to-use/Inputs/Outputs boilerplate + verbose LSP launch prose, compresses the dbc-overlapping hazards section) while preserving every rubric-rewarded behavior. Adds a 'trimmed' experiment arm and bumps runs to 3 so CI generates baseline/skilled/trimmed trajectories for local cross-family re-judging. Eval-only scratch branch; not part of the PR. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 3c9f9823-7f2f-4f7d-9d1b-f2b9e7a20c60

github-actions · 2026-07-17T18:39:13Z

👋 @AbhitejJohn — this PR has 1 unresolved review thread(s). When you're ready, please address the feedback and push an update; the triage bot will pick up the next state automatically. (Add the no-stale label to silence further pings.)

Robustness probe for the csharp-refactoring trim: all prior CI trajectories were executed by claude-opus-4.6. Flip executor to gpt-5.5 (GPT family) and CI judge to claude-opus-4.8 so the (executor, judge) pair stays cross-family. Tests whether trimmed-vs-current non-inferiority holds under a different model family. Eval-branch only; not on PR #873. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 3c9f9823-7f2f-4f7d-9d1b-f2b9e7a20c60

Closes the 'refactoring-only endpoint too narrow' risk flagged by the cross-family rubber-duck. Adds two boundary stimuli never used in trim tuning: 9. Separate a smuggled behavior change from a requested rename (rename + smuggled 10%->12% discount change) -> should separate/flag, not bundle. 10. Recognize a behavior-changing simplification is not a refactor (remove free-shipping-over-\ rule under a 'cleanup' framing) -> should not silently change observable behavior. Executor reverted to claude-opus-4.6 / judge gpt-5.5 (re-judge) to match the stringent Confirm A family where boundary behavior looked weakest. Eval-branch only; not on PR #873. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 3c9f9823-7f2f-4f7d-9d1b-f2b9e7a20c60

github-actions · 2026-07-21T18:46:38Z

👋 @AbhitejJohn — this PR has 1 unresolved review thread(s). When you're ready, please address the feedback and push an update; the triage bot will pick up the next state automatically. (Add the no-stale label to silence further pings.)

AbhitejJohn · 2026-07-22T06:13:00Z

@JanKrivanek — fair concern, thanks. I took the size + cross-loading cost seriously and did two things: trimmed both skills and ran a breadth eval to confirm the trims don't cost quality.

Size (SKILL.md bodies; the description frontmatter is unchanged, so activation/routing is held constant and only the in-context body shrank):

Skill	Before	After	Δ
`csharp-refactoring`	12,617 ch / 193 ln	5,953 ch / 85 ln	−53%
`dotnet-breaking-changes`	8,156 ch / 129 ln	6,425 ch / 98 ln	−21%

Combined the two bodies drop ~40% (20.8k→12.4k chars), which directly cuts the "load two long skills for one refactor" cost you flagged. I removed duplicated prose — the When to use / When NOT to use / Related skills lists (already encoded in the description's USE FOR / DO NOT USE FOR) and the overlapping "stop and escalate" / hidden-surface narration that was repeated across both skills — while keeping the inspect-first procedure, the four hidden-surface sections, escalation, and all reference files.

Eval — does the trim hurt? 5 executor families (opus-4.8, gpt-5.5, sonnet-4.6, haiku-4.5, mai-flash), a 2-family judge ensemble (gpt-5.5 + claude-opus-4.8), blinded arm order, n=5, with a negative-control scenario per skill:

vs no-skill: both trimmed skills post a non-negative quality delta on all 5 families under both judges; reliably positive (95% bootstrap CI excludes 0) on opus-4.8 and gpt-5.5, directional-positive/flat on the rest. Negative controls stay ~0 — the skills don't over-trigger on work that isn't a compatibility hazard.
vs the previous larger versions: no family is reliably degraded by the trim. For dotnet-breaking-changes, gpt-5.5 and haiku-4.5 actually favor the smaller version (+0.13/+0.16 mean, CI>0); opus/mai flat; sonnet slightly negative but inconclusive. For csharp-refactoring the trim also avoids a mild mai regression the larger version showed.

Honest caveats: single self-authored fixture suite, small scenario counts (6 for dbc), and CIs are bootstrapped over trials — so treat the sub-significant deltas as suggestive rather than proof of broad generalization. I'm not claiming formal non-inferiority, only that the smaller versions hold up at least as well as the larger ones across every family I could measure.

On the split: I kept it deliberately (breaking-changes is meant to apply to features/fixes, not only refactors), but the shared material is now much thinner, so cross-loading is meaningfully cheaper. Re-triggering /evaluate on the pushed trims below.

AbhitejJohn · 2026-07-22T06:13:07Z

/evaluate

Copilot

Copilot's findings

Files reviewed: 34/34 changed files
Comments generated: 2

github-actions · 2026-07-22T06:20:58Z

❌ Evaluation failed. View workflow run

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 8c45529b-2515-483d-9e51-e6c0b7cb6852

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 8c45529b-2515-483d-9e51-e6c0b7cb6852

- Updated the skill description to clarify usage and restrictions for refactoring requests. - Added new test cases for preserving serialized contracts in CustomerProfile. - Introduced CustomerProfile class to support new test scenarios.

…aming and compatibility

…ctoring skill documentation and tests

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

…elines and evaluation scenarios

…ntation for clarity and completeness

…ude ordinary tests

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Clarify mixed-request classification, add a real shipped-wrapper caller, and strengthen deterministic fixture and delegation guards. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 8c45529b-2515-483d-9e51-e6c0b7cb6852

Restore the focused one-operation workflow and remove broad worktree restore advice after integrating current main. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 8c45529b-2515-483d-9e51-e6c0b7cb6852

Copilot

Copilot review overview

🟡 Changes recommended

The missing advertised skill, incorrect experiment label, weak executable-test checks, and stale ownership entries must be addressed.

Get a fresh assessment by requesting another Copilot review.

Review tier: Lite
Findings: 1 · 1

Open (2)

Judge-comparison experiment label names the wrong judge · New
CODEOWNERS entries target a nonexistent skill · New

Move target discovery into a tested PowerShell script so GitHub Actions can parse and dispatch the workflow. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 8c45529b-2515-483d-9e51-e6c0b7cb6852

Copilot

Copilot review overview

🔵 Needs a closer look

The README/CODEOWNERS reference unshipped paths, and the extraction grader rejects valid equivalent refactorings.

Review tier: Lite
Findings: 1 · 1

Open (2)

Judge-comparison experiment label names the wrong judge
CODEOWNERS entries target a nonexistent skill

github-actions · 2026-09-15T17:29:17Z

📊 Skill and Agent Evaluation Results

2 model/target results across 1 target and 2 models — ✅ 2 improved, ➖ 0 not proven improved, ⚠️ 0 invalid or underpowered, ⛔ 0 activation contract failures, 📉 0 preference losses (report only).

Measurement identity: evaluated commit 1f73f159361e5c5a9d20164e00abb903518aea0d; 2 judge models.

Measurement health: 2 expected / 2 observed / 2 written; 0 missing, 0 unexpected, 0 invalid; 0 recovered comparison error slots and 0 unresolved comparison error slots.

Objective completion gate: not enabled. Aggregate completion transitions are telemetry only, so this report does not claim that zero objective regressions were proven.

A result passes only when preference-eligible distinct-stimulus votes have aggregate net win of at least 20% and an exact one-sided sign-test result of p ≤ 0.05, and every explicit dormancy activation contract passes. Repeated runs measure reliability only.

Target	Model	Verdict	Gate evidence	Overfit	Warnings	Next action
csharp-refactoring	claude-sonnet-5	✅ Improved	n=17; 11W/5T/1L; d=12; p=0.003; net +58.8%; 1 dormancy excluded	🟡 0.41	Activation: isolated 12/17; plugin 15/17	Fix activation gaps; Review overfit evidence.
csharp-refactoring	gpt-5.6-luna	✅ Improved	n=17; 8W/9T/0L; d=8; p=0.004; net +47.1%; 1 dormancy excluded	✅ 0.18	—	None.

ℹ️ How to read this report

✅ Improved — the result passed both the statistical gate and the 20% practical net-win floor.
➖ Not proven improved — the result is valid but did not pass both gates. This is not automatically a regression.
⚠️ Invalid / underpowered — the gate withheld a quality verdict. Fix the measurement before judging the target.
⛔ Activation contract failed — the isolated target activated on an explicit dormancy scenario. Dormancy preference is excluded, but this routing failure still blocks a pass.
📉 Preference loss — the LLM judge credibly preferred baseline. It is report-only, not objective completion proof.
Gate evidence — n preference-eligible distinct-stimulus votes, W/T/L stimulus votes, d discordant votes, exact one-sided p, net win, and the count of separately retained dormancy stimuli. The p value applies to one model/target result; no matrix-wide multiple-comparison correction is applied.
Overfit — overfitting-judge severity (✅ Low, 🟡 Moderate, 🔴 High, — none) and score.
Warnings — activation, timeout, retry recovery, or unresolved comparison conditions that need attention.
Do not add repeated runs to increase statistical power. Do not add stimuli after seeing a near-pass unless the new breadth is predeclared for a new experiment.

✅ Improved — csharp-refactoring (claude-sonnet-5)

Why: Net win +58.8% (11W/5T/1L over 17 preference-eligible stimulus vote(s), sign test p=0.003), mean preference +38.9% across 18 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — credibly better

Next action: Fix activation gaps; Review overfit evidence.

State: VALID_PASS (credible_preference_improvement)

Gate evidence: n=17; 11W/5T/1L; d=12; p=0.003; net +58.8%; 1 dormancy excluded

Warnings: Activation: isolated 12/17; plugin 15/17

Overfit: Moderate (score 0.41)

Repeated-run reliability (not used by the gate): 18 paired runs (11W/6T/1L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
▲ Consolidate duplicate public helpers without breaking shipped callers	Eligible	+100.0%	+100.0%	1/0/0
▲ Decline a framework and package upgrade dressed up as a refactor	Eligible	+100.0%	+100.0%	1/0/0
= Merge two near-identical types into one parameterized type	Eligible	+0.0%	+0.0%	0/1/0
= Preserve a shipped wrapper while inlining ordinary callers	Eligible	+0.0%	+0.0%	0/1/0
= Rename a helper used in every target-framework branch	Eligible	+0.0%	+0.0%	0/1/0
▼ Rename a member declared by a generated partial source	Eligible	-100.0%	-40.0%	0/0/1
= Rename a method without breaking a configured reflection name	Eligible	+0.0%	+0.0%	0/1/0
= Rename one binding without touching same-text symbols	Eligible	+0.0%	+0.0%	0/1/0
= Stay dormant for an ordinary feature request	Excluded (activation contract)	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

Merge two near-identical types into one parameterized type: Position-swap inconsistent (forward: B, reverse: tie). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

Routine passing details for 1 result are in Full Results.

🔍 Full Results - all metrics and investigation details

Correct the judge-comparison label and remove stale dotnet-breaking-changes registry entries left by the rebase. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 8c45529b-2515-483d-9e51-e6c0b7cb6852

Copilot

Copilot review overview

🔵 Needs a closer look

Unresolved findings cover inaccurate skill scope, an untested target-discovery path, and missing explicit build assertions.

Review tier: Lite
Findings: None

Resolved since last review (2)

Judge-comparison experiment label names the wrong judge
CODEOWNERS entries target a nonexistent skill

github-actions · 2026-09-15T17:58:40Z

📊 Skill and Agent Evaluation Results

2 model/target results across 1 target and 2 models — ✅ 2 improved, ➖ 0 not proven improved, ⚠️ 0 invalid or underpowered, ⛔ 0 activation contract failures, 📉 0 preference losses (report only).

Measurement identity: evaluated commit f4f0ed317a1315ddcde48092b5100cf57b2362f5; 2 judge models.

Measurement health: 2 expected / 2 observed / 2 written; 0 missing, 0 unexpected, 0 invalid; 0 recovered comparison error slots and 0 unresolved comparison error slots.

Objective completion gate: not enabled. Aggregate completion transitions are telemetry only, so this report does not claim that zero objective regressions were proven.

A result passes only when preference-eligible distinct-stimulus votes have aggregate net win of at least 20% and an exact one-sided sign-test result of p ≤ 0.05, and every explicit dormancy activation contract passes. Repeated runs measure reliability only.

Target	Model	Verdict	Gate evidence	Overfit	Warnings	Next action
csharp-refactoring	claude-sonnet-5	✅ Improved	n=17; 12W/4T/1L; d=13; p=0.002; net +64.7%; 1 dormancy excluded	🟡 0.41	Activation: isolated 14/17; plugin 13/17	Fix activation gaps; Review overfit evidence.
csharp-refactoring	gpt-5.6-luna	✅ Improved	n=17; 7W/9T/1L; d=8; p=0.035; net +35.3%; 1 dormancy excluded	🟡 0.22	—	Review overfit evidence.

ℹ️ How to read this report

✅ Improved — the result passed both the statistical gate and the 20% practical net-win floor.
➖ Not proven improved — the result is valid but did not pass both gates. This is not automatically a regression.
⚠️ Invalid / underpowered — the gate withheld a quality verdict. Fix the measurement before judging the target.
⛔ Activation contract failed — the isolated target activated on an explicit dormancy scenario. Dormancy preference is excluded, but this routing failure still blocks a pass.
📉 Preference loss — the LLM judge credibly preferred baseline. It is report-only, not objective completion proof.
Gate evidence — n preference-eligible distinct-stimulus votes, W/T/L stimulus votes, d discordant votes, exact one-sided p, net win, and the count of separately retained dormancy stimuli. The p value applies to one model/target result; no matrix-wide multiple-comparison correction is applied.
Overfit — overfitting-judge severity (✅ Low, 🟡 Moderate, 🔴 High, — none) and score.
Warnings — activation, timeout, retry recovery, or unresolved comparison conditions that need attention.
Do not add repeated runs to increase statistical power. Do not add stimuli after seeing a near-pass unless the new breadth is predeclared for a new experiment.

✅ Improved — csharp-refactoring (claude-sonnet-5)

Why: Net win +64.7% (12W/4T/1L over 17 preference-eligible stimulus vote(s), sign test p=0.002), mean preference +34.4% across 18 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — credibly better

Next action: Fix activation gaps; Review overfit evidence.

State: VALID_PASS (credible_preference_improvement)

Gate evidence: n=17; 12W/4T/1L; d=13; p=0.002; net +64.7%; 1 dormancy excluded

Warnings: Activation: isolated 14/17; plugin 13/17

Overfit: Moderate (score 0.41)

Repeated-run reliability (not used by the gate): 18 paired runs (12W/5T/1L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Extract shared discount logic without collapsing distinct inputs	Eligible	+0.0%	+0.0%	0/1/0
= Keep a shipped nullable contract out of a refactor	Eligible	+0.0%	+0.0%	0/1/0
= Preserve a serialized name while renaming a C# property	Eligible	+0.0%	+0.0%	0/1/0
▼ Preserve a shipped wrapper while inlining ordinary callers	Eligible	-100.0%	-100.0%	0/0/1
▲ Rename a helper used in every target-framework branch	Eligible	+100.0%	+40.0%	1/0/0
▲ Rename a member declared by a generated partial source	Eligible	+100.0%	+40.0%	1/0/0
▲ Rename a method without breaking a configured reflection name	Eligible	+100.0%	+40.0%	1/0/0
= Rename a shipped public method without breaking existing callers	Eligible	+0.0%	+0.0%	0/1/0
= Stay dormant for an ordinary feature request	Excluded (activation contract)	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

Extract shared discount logic without collapsing distinct inputs: Position-swap inconsistent (forward: tie, reverse: A). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

✅ Improved — csharp-refactoring (gpt-5.6-luna)

Why: Net win +35.3% (7W/9T/1L over 17 preference-eligible stimulus vote(s), sign test p=0.035), mean preference +23.3% across 18 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — credibly better

Next action: Review overfit evidence.

State: VALID_PASS (credible_preference_improvement)

Gate evidence: n=17; 7W/9T/1L; d=8; p=0.035; net +35.3%; 1 dormancy excluded

Overfit: Moderate (score 0.22)

Repeated-run reliability (not used by the gate): 18 paired runs (7W/10T/1L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Consolidate duplicate public helpers without breaking shipped callers	Eligible	+0.0%	+0.0%	0/1/0
= Extract subtotal calculation without changing arithmetic	Eligible	+0.0%	+0.0%	0/1/0
= Inline a pass-through wrapper and update callers	Eligible	+0.0%	+0.0%	0/1/0
▼ Merge two near-identical types into one parameterized type	Eligible	-100.0%	-100.0%	0/0/1
= Preserve a serialized name while renaming a C# property	Eligible	+0.0%	+0.0%	0/1/0
= Preserve a shipped wrapper while inlining ordinary callers	Eligible	+0.0%	+0.0%	0/1/0
= Rename a member declared by a generated partial source	Eligible	+0.0%	+0.0%	0/1/0
= Rename a method without breaking a configured reflection name	Eligible	+0.0%	+0.0%	0/1/0
= Rename an internal member used by a friend assembly	Eligible	+0.0%	+0.0%	0/1/0
= Rename one binding without touching same-text symbols	Eligible	+0.0%	+0.0%	0/1/0
= Stay dormant for an ordinary feature request	Excluded (activation contract)	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

Consolidate duplicate public helpers without breaking shipped callers: Position-swap inconsistent (forward: B, reverse: tie). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

🔍 Full Results - all metrics and investigation details

github-actions · 2026-09-15T18:31:19Z

✅ Evaluation passed for f4f0ed3. cc @webreidi @AbhitejJohn @JanKrivanek — please review.

github-actions · 2026-09-15T21:08:49Z

📊 Skill and Agent Evaluation Results

6 model/target results across 1 target and 6 models — ✅ 4 improved, ➖ 2 not proven improved, ⚠️ 0 invalid or underpowered, ⛔ 0 activation contract failures, 📉 0 preference losses (report only).

Measurement identity: evaluated commit f4f0ed317a1315ddcde48092b5100cf57b2362f5; 2 judge models.

Measurement health: 6 expected / 6 observed / 6 written; 0 missing, 0 unexpected, 0 invalid; 0 recovered comparison error slots and 0 unresolved comparison error slots.

Objective completion gate: not enabled. Aggregate completion transitions are telemetry only, so this report does not claim that zero objective regressions were proven.

A result passes only when preference-eligible distinct-stimulus votes have aggregate net win of at least 20% and an exact one-sided sign-test result of p ≤ 0.05, and every explicit dormancy activation contract passes. Repeated runs measure reliability only.

Target	Model	Verdict	Gate evidence	Overfit	Warnings	Next action
csharp-refactoring	claude-haiku-4.5	✅ Improved	n=17; 5W/12T/0L; d=5; p=0.031; net +29.4%; 1 dormancy excluded	🟡 0.41	Activation: isolated 12/17; plugin 14/17	Fix activation gaps; Review overfit evidence.
csharp-refactoring	claude-opus-4.8	✅ Improved	n=17; 10W/7T/0L; d=10; p=0.001; net +58.8%; 1 dormancy excluded	🟡 0.50	Activation: isolated 14/17; plugin 16/17	Fix activation gaps; Review overfit evidence.
csharp-refactoring	claude-sonnet-5	✅ Improved	n=17; 14W/3T/0L; d=14; p=0.000; net +82.4%; 1 dormancy excluded	🟡 0.48	Activation: isolated 15/17; plugin 13/17	Fix activation gaps; Review overfit evidence.
csharp-refactoring	gpt-5.3-codex	➖ Not proven improved	n=17; 7W/6T/4L; d=11; p=0.274; net +17.6%; 1 dormancy excluded	✅ 0.19	Activation: isolated 7/17; plugin 8/17; Activation-only stop: isolated 1 failed run; Activation-only stop: plugin 2 failed runs	Inspect activation-only failed runs before rewriting skill content; the model stopped after loading a skill.
csharp-refactoring	gpt-5.6-luna	✅ Improved	n=17; 8W/8T/1L; d=9; p=0.020; net +41.2%; 1 dormancy excluded	✅ 0.17	—	None.
csharp-refactoring	mai-code-1.1-flash	➖ Not proven improved	n=17; 4W/11T/2L; d=6; p=0.344; net +11.8%; 1 dormancy excluded	🟡 0.31	Activation: isolated 2/17; plugin 0/17	Inspect tied or lost stimuli and fix inconsistent skill behavior.

ℹ️ How to read this report

✅ Improved — the result passed both the statistical gate and the 20% practical net-win floor.
➖ Not proven improved — the result is valid but did not pass both gates. This is not automatically a regression.
⚠️ Invalid / underpowered — the gate withheld a quality verdict. Fix the measurement before judging the target.
⛔ Activation contract failed — the isolated target activated on an explicit dormancy scenario. Dormancy preference is excluded, but this routing failure still blocks a pass.
📉 Preference loss — the LLM judge credibly preferred baseline. It is report-only, not objective completion proof.
Gate evidence — n preference-eligible distinct-stimulus votes, W/T/L stimulus votes, d discordant votes, exact one-sided p, net win, and the count of separately retained dormancy stimuli. The p value applies to one model/target result; no matrix-wide multiple-comparison correction is applied.
Overfit — overfitting-judge severity (✅ Low, 🟡 Moderate, 🔴 High, — none) and score.
Warnings — activation, timeout, retry recovery, or unresolved comparison conditions that need attention.
Do not add repeated runs to increase statistical power. Do not add stimuli after seeing a near-pass unless the new breadth is predeclared for a new experiment.

➖ Not proven improved — csharp-refactoring (gpt-5.3-codex)

Why: Net win +17.6% (7W/6T/4L over 17 preference-eligible stimulus vote(s), sign test p=0.274), mean preference +6.7% across 18 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — not credible (sign test p=0.274 > 0.05)

Next action: Inspect activation-only failed runs before rewriting skill content; the model stopped after loading a skill.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=17; 7W/6T/4L; d=11; p=0.274; net +17.6%; 1 dormancy excluded

Warnings: Activation: isolated 7/17; plugin 8/17; Activation-only stop: isolated 1 failed run; Activation-only stop: plugin 2 failed runs

Overfit: Low (score 0.19)

Repeated-run reliability (not used by the gate): 18 paired runs (7W/7T/4L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
▲ Consolidate duplicate public helpers without breaking shipped callers	Eligible	+100.0%	+40.0%	1/0/0
▲ Decline a framework and package upgrade dressed up as a refactor	Eligible	+100.0%	+100.0%	1/0/0
= Extract shared discount logic without collapsing distinct inputs	Eligible	+0.0%	+0.0%	0/1/0
▼ Extract subtotal calculation without changing arithmetic	Eligible	-100.0%	-100.0%	0/0/1
= Inline a pass-through wrapper and update callers	Eligible	+0.0%	+0.0%	0/1/0
= Keep a shipped nullable contract out of a refactor	Eligible	+0.0%	+0.0%	0/1/0
▲ Keep behavior-changing bug fixes out of a behavior-preserving refactor	Eligible	+100.0%	+100.0%	1/0/0
= Merge two near-identical types into one parameterized type	Eligible	+0.0%	+0.0%	0/1/0
▼ Preserve a shipped wrapper while inlining ordinary callers	Eligible	-100.0%	-40.0%	0/0/1
= Rename a helper used in every target-framework branch	Eligible	+0.0%	+0.0%	0/1/0
▼ Rename a member declared by a generated partial source	Eligible	-100.0%	-100.0%	0/0/1
= Rename a method without breaking a configured reflection name	Eligible	+0.0%	+0.0%	0/1/0
▼ Rename an internal member used by a friend assembly	Eligible	-100.0%	-100.0%	0/0/1
▲ Rename one binding without touching same-text symbols	Eligible	+100.0%	+40.0%	1/0/0
= Stay dormant for an ordinary feature request	Excluded (activation contract)	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

Extract shared discount logic without collapsing distinct inputs: Position-swap inconsistent (forward: tie, reverse: B). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — csharp-refactoring (mai-code-1.1-flash)

Why: Net win +11.8% (4W/11T/2L over 17 preference-eligible stimulus vote(s), sign test p=0.344), mean preference +10.0% across 18 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — not credible (sign test p=0.344 > 0.05)

Next action: Inspect tied or lost stimuli and fix inconsistent skill behavior.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=17; 4W/11T/2L; d=6; p=0.344; net +11.8%; 1 dormancy excluded

Warnings: Activation: isolated 2/17; plugin 0/17

Overfit: Moderate (score 0.31)

Repeated-run reliability (not used by the gate): 18 paired runs (5W/11T/2L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
▲ Consolidate duplicate public helpers without breaking shipped callers	Eligible	+100.0%	+40.0%	1/0/0
= Decline a framework and package upgrade dressed up as a refactor	Eligible	+0.0%	+0.0%	0/1/0
= Decline a new-feature request dressed up as a refactor	Eligible	+0.0%	+0.0%	0/1/0
▲ Extract shared discount logic without collapsing distinct inputs	Eligible	+100.0%	+40.0%	1/0/0
= Extract subtotal calculation without changing arithmetic	Eligible	+0.0%	+0.0%	0/1/0
= Inline a pass-through wrapper and update callers	Eligible	+0.0%	+0.0%	0/1/0
= Keep a shipped nullable contract out of a refactor	Eligible	+0.0%	+0.0%	0/1/0
▲ Keep behavior-changing bug fixes out of a behavior-preserving refactor	Eligible	+100.0%	+40.0%	1/0/0
= Merge two near-identical types into one parameterized type	Eligible	+0.0%	+0.0%	0/1/0
= Preserve a serialized name while renaming a C# property	Eligible	+0.0%	+0.0%	0/1/0
= Preserve a shipped wrapper while inlining ordinary callers	Eligible	+0.0%	+0.0%	0/1/0
= Rename a helper used in every target-framework branch	Eligible	+0.0%	+0.0%	0/1/0
= Rename a member declared by a generated partial source	Eligible	+0.0%	+0.0%	0/1/0
▼ Rename a method without breaking a configured reflection name	Eligible	-100.0%	-40.0%	0/0/1
= Rename a shipped public method without breaking existing callers	Eligible	+0.0%	+0.0%	0/1/0
▲ Rename an internal member used by a friend assembly	Eligible	+100.0%	+40.0%	1/0/0
▼ Rename one binding without touching same-text symbols	Eligible	-100.0%	-40.0%	0/0/1

Illustrative judge evidence:

Decline a framework and package upgrade dressed up as a refactor: Position-swap inconsistent (forward: A, reverse: tie). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

✅ Improved — csharp-refactoring (claude-haiku-4.5)

Why: Net win +29.4% (5W/12T/0L over 17 preference-eligible stimulus vote(s), sign test p=0.031), mean preference +15.6% across 18 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — credibly better

Next action: Fix activation gaps; Review overfit evidence.

State: VALID_PASS (credible_preference_improvement)

Gate evidence: n=17; 5W/12T/0L; d=5; p=0.031; net +29.4%; 1 dormancy excluded

Warnings: Activation: isolated 12/17; plugin 14/17

Overfit: Moderate (score 0.41)

Repeated-run reliability (not used by the gate): 18 paired runs (5W/12T/1L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Consolidate duplicate public helpers without breaking shipped callers	Eligible	+0.0%	+0.0%	0/1/0
= Extract shared discount logic without collapsing distinct inputs	Eligible	+0.0%	+0.0%	0/1/0
= Extract subtotal calculation without changing arithmetic	Eligible	+0.0%	+0.0%	0/1/0
= Inline a pass-through wrapper and update callers	Eligible	+0.0%	+0.0%	0/1/0
▲ Keep a shipped nullable contract out of a refactor	Eligible	+100.0%	+40.0%	1/0/0
= Keep behavior-changing bug fixes out of a behavior-preserving refactor	Eligible	+0.0%	+0.0%	0/1/0
= Merge two near-identical types into one parameterized type	Eligible	+0.0%	+0.0%	0/1/0
= Preserve a serialized name while renaming a C# property	Eligible	+0.0%	+0.0%	0/1/0
= Preserve a shipped wrapper while inlining ordinary callers	Eligible	+0.0%	+0.0%	0/1/0
▲ Rename a helper used in every target-framework branch	Eligible	+100.0%	+40.0%	1/0/0
= Rename a method without breaking a configured reflection name	Eligible	+0.0%	+0.0%	0/1/0
= Rename a shipped public method without breaking existing callers	Eligible	+0.0%	+0.0%	0/1/0
= Rename an internal member used by a friend assembly	Eligible	+0.0%	+0.0%	0/1/0
= Rename one binding without touching same-text symbols	Eligible	+0.0%	+0.0%	0/1/0
▼ Stay dormant for an ordinary feature request	Excluded (activation contract)	-100.0%	-40.0%	0/0/1

Illustrative judge evidence:

Consolidate duplicate public helpers without breaking shipped callers: Position-swap inconsistent (forward: tie, reverse: A). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

✅ Improved — csharp-refactoring (claude-opus-4.8)

Why: Net win +58.8% (10W/7T/0L over 17 preference-eligible stimulus vote(s), sign test p=0.001), mean preference +38.9% across 18 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — credibly better

Next action: Fix activation gaps; Review overfit evidence.

State: VALID_PASS (credible_preference_improvement)

Gate evidence: n=17; 10W/7T/0L; d=10; p=0.001; net +58.8%; 1 dormancy excluded

Warnings: Activation: isolated 14/17; plugin 16/17

Overfit: Moderate (score 0.50)

Repeated-run reliability (not used by the gate): 18 paired runs (10W/8T/0L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
▲ Decline a framework and package upgrade dressed up as a refactor	Eligible	+100.0%	+100.0%	1/0/0
▲ Extract shared discount logic without collapsing distinct inputs	Eligible	+100.0%	+40.0%	1/0/0
= Extract subtotal calculation without changing arithmetic	Eligible	+0.0%	+0.0%	0/1/0
= Inline a pass-through wrapper and update callers	Eligible	+0.0%	+0.0%	0/1/0
= Merge two near-identical types into one parameterized type	Eligible	+0.0%	+0.0%	0/1/0
= Preserve a serialized name while renaming a C# property	Eligible	+0.0%	+0.0%	0/1/0
= Rename a helper used in every target-framework branch	Eligible	+0.0%	+0.0%	0/1/0
= Rename a member declared by a generated partial source	Eligible	+0.0%	+0.0%	0/1/0
= Rename a method without breaking a configured reflection name	Eligible	+0.0%	+0.0%	0/1/0
= Stay dormant for an ordinary feature request	Excluded (activation contract)	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

Extract subtotal calculation without changing arithmetic: Position-swap inconsistent (forward: B, reverse: tie). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

✅ Improved — csharp-refactoring (claude-sonnet-5)

Why: Net win +82.4% (14W/3T/0L over 17 preference-eligible stimulus vote(s), sign test p=0.000), mean preference +47.8% across 18 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — credibly better

Next action: Fix activation gaps; Review overfit evidence.

State: VALID_PASS (credible_preference_improvement)

Gate evidence: n=17; 14W/3T/0L; d=14; p=0.000; net +82.4%; 1 dormancy excluded

Warnings: Activation: isolated 15/17; plugin 13/17

Overfit: Moderate (score 0.48)

Repeated-run reliability (not used by the gate): 18 paired runs (14W/4T/0L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Consolidate duplicate public helpers without breaking shipped callers	Eligible	+0.0%	+0.0%	0/1/0
▲ Decline a framework and package upgrade dressed up as a refactor	Eligible	+100.0%	+100.0%	1/0/0
= Preserve a serialized name while renaming a C# property	Eligible	+0.0%	+0.0%	0/1/0
▲ Preserve a shipped wrapper while inlining ordinary callers	Eligible	+100.0%	+40.0%	1/0/0
▲ Rename a helper used in every target-framework branch	Eligible	+100.0%	+40.0%	1/0/0
▲ Rename a method without breaking a configured reflection name	Eligible	+100.0%	+40.0%	1/0/0
= Rename an internal member used by a friend assembly	Eligible	+0.0%	+0.0%	0/1/0
= Stay dormant for an ordinary feature request	Excluded (activation contract)	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

Consolidate duplicate public helpers without breaking shipped callers: The substantive change and verification are equivalent and satisfy the task. B's final write-up is somewhat more detailed, but it does not establish a materially better implementation or validation result.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

Routine passing details for 1 result are in Full Results.

🔍 Full Results - all metrics and investigation details

To investigate non-passing or warning results, paste this to your AI coding agent:

For PR 873 in dotnet/skills, download eval artifacts with gh run download 35019805981 --repo dotnet/skills --pattern "vally-results-*" --dir ./eval-results, then fetch https://raw.githubusercontent.com/dotnet/skills/f4f0ed317a1315ddcde48092b5100cf57b2362f5/eng/vally-adapter/InvestigatingResults.md and follow it. Classify each result as measurement-invalid, underpowered, not-proven, preference-loss, or passing-with-warning. Use stateReason, result accounting, weak scenarios, and judge evidence to give the cause and exact next fix.

github-actions Bot added a commit that referenced this pull request Jul 8, 2026

Update PR token usage data (PR #873)

7725366

AbhitejJohn marked this pull request as ready for review July 13, 2026 17:18

Copilot AI lite review requested due to automatic review settings July 13, 2026 17:18

AbhitejJohn requested review from JanKrivanek, ViktorHofer and webreidi as code owners July 13, 2026 17:18

Copilot started reviewing on behalf of AbhitejJohn July 13, 2026 17:19 View session

Copilot AI reviewed Jul 13, 2026

View reviewed changes

webreidi reviewed Jul 13, 2026

View reviewed changes

Comment thread plugins/dotnet/skills/csharp-refactoring/SKILL.md Outdated

github-actions Bot added the waiting-on-author PR state label label Jul 13, 2026

JanKrivanek reviewed Jul 17, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings July 22, 2026 06:10

Copilot started reviewing on behalf of AbhitejJohn July 22, 2026 06:11 View session

Copilot AI reviewed Jul 22, 2026

View reviewed changes

Comment thread tests/dotnet/dotnet-breaking-changes/src/Billing/Billing.csproj Outdated

Comment thread tests/dotnet/csharp-refactoring/src/Billing/Billing.csproj

Copilot AI review requested due to automatic review settings July 23, 2026 07:10

Copilot started reviewing on behalf of AbhitejJohn July 23, 2026 07:11 View session

AbhitejJohn and others added 18 commits September 15, 2026 09:18

Fix csharp refactoring eval fixtures

5c80132

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 8c45529b-2515-483d-9e51-e6c0b7cb6852

Harden csharp refactoring eval graders

e03bcfb

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 8c45529b-2515-483d-9e51-e6c0b7cb6852

Add refactoring dormancy coverage

a9b8c41

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 8c45529b-2515-483d-9e51-e6c0b7cb6852

Remove redundant refactoring eval checks

040d33f

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 8c45529b-2515-483d-9e51-e6c0b7cb6852

Add file-contains check for Shipped.txt

31290d2

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Refactor ConfigReader body extraction method

c797c40

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Scope ConfigReader grader to its class

d450573

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 8c45529b-2515-483d-9e51-e6c0b7cb6852

Refactor operation catalog and update evaluation tests for method ren…

31ca050

…aming and compatibility

Update experiment name in evaluation workflow and enhance csharp-refa…

72fc029

…ctoring skill documentation and tests

Update SKILL.md with mandatory gate clarification

7f0b539

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Enhance csharp-refactoring skill documentation with new decision guid…

63c0d73

…elines and evaluation scenarios

Refine mandatory gate instructions in csharp-refactoring skill docume…

2c4ed80

…ntation for clarity and completeness

Fix compatibility test logic in csharp-refactoring evaluation to excl…

d016cf9

…ude ordinary tests

Revise refactoring and test failure guidelines

11870a3

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Update SKILL.md with clearer refactoring guidelines

3c8cab0

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Copilot AI reviewed Sep 15, 2026

View reviewed changes

Comment thread .github/workflows/evaluation.yml Outdated

Comment thread .github/CODEOWNERS Outdated

Fix evaluation workflow expression limit

1f73f15

Move target discovery into a tested PowerShell script so GitHub Actions can parse and dispatch the workflow. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 8c45529b-2515-483d-9e51-e6c0b7cb6852

Copilot AI reviewed Sep 15, 2026

View reviewed changes

Address refactoring PR review feedback

f4f0ed3

Correct the judge-comparison label and remove stale dotnet-breaking-changes registry entries left by the rebase. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 8c45529b-2515-483d-9e51-e6c0b7cb6852

Copilot AI reviewed Sep 15, 2026

View reviewed changes

webreidi approved these changes Sep 15, 2026

View reviewed changes

AbhitejJohn mentioned this pull request Sep 16, 2026

Fix csharp-refactoring routing and eval warnings #1185

Open

github-actions Bot mentioned this pull request Sep 21, 2026

Weekly plugin version sync #1194

Open

Conversation

AbhitejJohn commented Jul 8, 2026

Summary

Why these skills

Validation

Uh oh!

github-actions Bot commented Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Skill Coverage Report

Uh oh!

AbhitejJohn commented Jul 8, 2026

Uh oh!

github-actions Bot commented Jul 8, 2026

Skill Validation Results

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Copilot's findings

Uh oh!

AbhitejJohn commented Jul 13, 2026

Uh oh!

Uh oh!

github-actions Bot commented Jul 13, 2026

Uh oh!

github-actions Bot commented Jul 13, 2026

Skill Validation Results

Uh oh!

JanKrivanek left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Jul 17, 2026

Uh oh!

github-actions Bot commented Jul 21, 2026

Uh oh!

AbhitejJohn commented Jul 22, 2026

Uh oh!

AbhitejJohn commented Jul 22, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot's findings

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Jul 22, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🔵 Needs a closer look

Uh oh!

github-actions Bot commented Sep 15, 2026

📊 Skill and Agent Evaluation Results

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🔵 Needs a closer look

Uh oh!

github-actions Bot commented Sep 15, 2026

📊 Skill and Agent Evaluation Results

Uh oh!

github-actions Bot commented Sep 15, 2026

Uh oh!

github-actions Bot commented Sep 15, 2026

📊 Skill and Agent Evaluation Results

Uh oh!

This branch was successfully deployed

Reviewers

Assignees

Labels

Projects

github-actions Bot commented Jul 8, 2026 •

edited

Loading